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Managing  an  acquisition  program  in  the  DoD  is  a 
complicated  process.  The  turbulence  created  by  funding 
instability  can  make  it  even  more  difficult.  Nonetheless, 
to  help  program  offices  maintain  their  overall  funding 
execution  pace,  the  Office  of  the  Secretary  of  Defense 
(OSD)  instituted  Obligation  and  Expenditure  rate  goals 
over  two  decades  ago.  For  numerous  reasons,  acquisi¬ 
tion  program  managers  have  found  it  difficult  to  meet 
established  Obligation  and  Expenditure  rate  goals.  For 
purposes  of  this  article,  and  based  on  Defense  Acquisi¬ 
tion  University  and  OSD  subject  matter  expertise,  the 
authors  looked  more  closely  at  the  potential  causal 
factors  that  could  be  interfering  with  the  achievement 
of  these  goals. 
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Several  months  ago.  Dr.  Nancy  Spruill,  director  of  Acquisition 
Resources  and  Analysis,  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics,  solicited  support  from  the  Defense 
Acquisition  University  (DAU)  to  help  uncover  the  causal  factors  that  could 
be  interfering  with  the  attainment  of  OSD’s  Obligation  and  Expenditure 
rate  goals.  To  learn  more  about  the  intervening  obstacles,  DAU,  with 
assistance  from  OSD,  developed  a  comprehensive  survey  that  queried 
experienced  and  high-level  DoD  personnel  involved  in  a  weapon  pro¬ 
gram’s  decision  chain.  The  data  might  also  indicate  the  prevalence  of  any 
significant  variances  among  the  factors  that  could  be  undermining  pro¬ 
gram  execution  itself.  Results  of  the  study  (Higbee,  Tremaine,  Seligman, 
&  Arwood,  2013)  were  presented  to  Assistant  Secretary  of  Defense  for 
Acquisition  Katrina  McFarland  and  other  senior  OSD  personnel. 

Research  Methodology 

Two  hundred  and  twenty-nine  DoD  personnel  responded  to  this 
survey.  The  respondents  were  comprised  of  program  office  personnel 
(program  managers,  deputy  program  managers,  budget  and  financial 
managers,  and  contracting  officers);  program  executive  officers  and  their 
chief  financial  officers;  and  a  variety  of  senior  staff  at  OSD  including 
Headquarters  Financial  Management  senior  staff  and  Senior  Acquisition 
Executive  (SAE)  staff  (Table  1).  Because  several  functional  areas 
reflected  lower  response  rates,  a  more  detailed  analysis  of  the  causal 
factors  was  restricted  to  an  aggregate  sample  size  given  the  confidence 
levels  required  to  draw  any  inferences  or  conclusions. 

TABLE  1.  INDIVIDUAL  RESPONDENT  GROUPS 


Survey  Respondent  Details 


ACAT  Levels 

Respondent  Groups 

Totals 

Respondent 

Program 

Senior 

Response 

Distribution3 

1  II 

III 

Officeb 

PEOc 

Staffd 

Responses 

Queried 

Rate 

Total 

91  28 

23 

142 

63 

24 

229 

698 

33% 

a  Includes  sampling  from  all  DoD  Components  and  several  Defense  Agencies 

b  Program  managers,  deputy  program  managers,  business-financial  management  (BFM)  managers, 
deputy  BFM  managers,  and  contracting  officers 
c  Program  executive  officers  (PEO),  deputy  PEOs,  and  their  chief  financial  officers 
d  Headquarters,  Financial  Management  and  Senior  Acquisition  Executive  staff 
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Respondents  ranked  the  impact  of  64  factors  under  nine  categories 
(Figure  1).  The  researchers  then  assessed  the  rankings  using  atop  box 
(TB)  three  methodology  (i.e.,  the  percentage  of  5,  6,  and  7  responses  on  a 
Likert-like  scale  from  1-7).  Since  the  frequency  of  occurrence  for  some 
factors  could  also  be  contributing  to  the  interference,  the  researchers 
included  an  additional  selection  (e.g.,  daily,  weekly,  monthly,  etc.)  to 
isolate  any  potential  ignition  areas  for  any  factor. 


FIGURE  1.  FACTOR  CATEGORIES 


Defense ARJ,  October 3013,  Vol.  20 No.  3: 373-400 


376 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


Discussion 


Factor  Distribution 

Figure  2  shows  the  distribution  of  all  64  factors  assessed.  Three 
factors  reported  an  impact  rating  of  two  standard  deviations  above 
the  mean  (denoted  by  +2er);  six  factors  reported  an  impact  rating  of  one 
standard  deviation  above  the  mean  (denoted  by  +ler);  and  22  factors  rose 
above  an  average  impact  rating  (denoted  by  x).  The  remaining  33  factors 
fell  below  x. 

FIGURE  2.  RESPONDENT  HISTOGRAM 


IMPACT  FACTOR  DISTRIBUTION 


Impact  Ratings 


Nineteen  of  the  22  factors  measured  for  frequency  of  occurrence 
resulted  in  an  impact  rating  above  39  percent.  Sometimes  Just  one  occur¬ 
rence  of  that  factor  appeared  to  have  a  significant  impact. 

Causal  Factors  Rank  Ordered 

Table  2  lists  the  relative  ranking  of  all  64  factors  in  the  context  of 
TB  in  descending  order.  This  ranking  provides  a  comprehensive  view  of 
all  factors  although  the  remaining  discussion  in  this  article  addresses 
only  the  factors  above  x.  One  particular  factor,  “Unrealistic,  overly 
optimistic  spend  plans”  (F10),  is  important  to  note  since  it  serves  as  a 
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written  forecast  of  a  program’s  funding  needs  and  initially  establishes 
Obligation  and  Expenditure  projections.  However,  spend  plans  are  also 
subjected  to  so  many  real  world  eventualities  that  updating  them  can 
become  a  full-time  job. 

Factors  and  Respondent  Groups 

Figure  3  accounts  for  the  31  factors  above  the  mean  and  by  respon¬ 
dent  group  as  depicted  in  Table  1.  The  31  factors  were  the  only  ones 
further  evaluated  in  this  study  unless  a  factor  shifted  above  x  after  a 
more  detailed  correlation  delineation  (e.g.,  Acquisition  Category  [ACAT]) 
levels,  military  components,  position,  etc.).  Unexpectedly,  the  individual 
factors  showed  widespread  perception  disparities  among  the  respondent 
groups  for  the  factors  that  fell  below  +2er.  After  analyzing  the  specific 
individual  factors  among  all  the  respondent  groups,  seven  of  the  31  fac¬ 
tors  had  an  unusually  large  a.  As  a  result  of  these  conspicuous  gaps,  the 
authors  turned  to  the  qualitative  data  and  watched  for  any  strong  cor¬ 
relations  (e.g.,  positive  quantitative  correlation  coefficients  (r)  >  0.7)  to 
better  understand  the  reasons  for  the  differences  as  well  as  the  influence 
of  any  intervening  and/or  moderating  factor  couplings.  The  remaining 
discussion  addresses  the  31  impact  factors  in  descending  order  from 
highest  to  lowest. 

Factors  Ranked  Two  Standard  Deviations  Above  the 
Mean  (+  2d) 

In  Figure  3,  Late  release  of  full  obligation/budget  authority  due  to 
Continuing  Resolution  Authority  (CRA)  (F1),  Contract  negotiations’ 
delays  (F2),  and  Contract  award  delays  (F3)  all  rose  above  2er  where  67 
percent  or  more  of  the  respondents  claimed  they  had  the  highest  adverse 
impact  of  all  factors  measured.  The  occurrence  of  CRA  had  the  most 
significant  negative  impact  to  Obligation  and  Expenditure  rates.  It  also 
had  one  of  the  smallest  variances  (a)  among  the  respondent  groups.  Even 
with  the  expectation  that  CRA  might  prevail  and  the  subsequent  plan¬ 
ning  that  followed  for  such  a  likely  event,  many  PMs  pointed  to  an  overly 
conservative  and  slow  internal  vetting  process  posture  that  created  addi¬ 
tional  obstacles  in  meeting  OSD  goals.  In  their  responses  to  qualitative 
questions,  several  PMs  recommended  using  some  sort  of  “CRA  variable” 
to  temporarily  offset  the  consequences  of  CRA  if  the  required  funds  were 
not  released  as  originally  projected.  Next  in  rank  order  were  contract 
negotiations  and  contract  award  delays.  The  respondents  emphasized 
that  DoD  could  fix  the  problem  more  readily  since  unlike  CRA,  these 
factors  were  under  internal  control.  When  asked  what  could  be  done 
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TABLE  2.  IMPACT  FACTOR  RATINGS  IN  AGGREGATE  DESCENDING  ORDER 


Factors  Rated  by  Adverse  Impact 

TB 

X 

a 

F1 

Late  release  of  full  obligation  budget  authority  due  to  Continuing 
Resolution  Authority  (CRA) 

69% 

5.29 

2.41 

F2 

Contract  negotiation  delays 

67% 

5.06 

2.59 

F3 

Contract  award  delays 

67% 

5.00 

2.56 

F4 

Shortage  of  contracting  officers 

64% 

4.79 

2.58 

F5 

Congressional  mark/rescission 

61% 

4.87 

2.65 

F6 

Contractor  proposal  prep  delays 

60% 

4.87 

2.59 

F7 

OSD-directed  Resource  Management  Decision  (RMD) 

58% 

4.50 

2.63 

FB 

Request  for  Proposal  (RFP)  prep  delays 

57% 

4.63 

2.46 

F9 

Source  selection  delays 

55% 

4.44 

2.53 

F!0 

Unrealistic,  overly  optimistic  spend  plans 

52% 

4.30 

2.44 

F" 

Changes  in  user  requirements 

51% 

4.16 

2.43 

F'2 

Changes  to  program  acquisition  strategy 

51% 

4.41 

2.52 

F'3 

Changes  in  other  stakeholder  requirements 

50% 

4.32 

2.34 

F'4 

Preparing  Defense  Acquisition  Executive  (DAE)-level  review 
and  decision 

50% 

4.15 

2.18 

F’5 

Lack  of  decision  authority  at  expected  levels 

50% 

4.22 

2.52 

F’6 

Implementation  of  new  OSD/Service  policy 

49% 

4.20 

2.59 

F’7 

Component-directed  Program  Objective  Memorandum 
(POM)  adjustment 

49% 

4.26 

2.51 

F18 

Awaiting  reprogramming  action 

49% 

4.23 

2.44 

F’9 

Changes  in  user  priorities 

47% 

4.00 

2.38 

p20 

Realistic  spend  plans,  but  risks  materialized 

45% 

4.00 

2.21 

F2' 

Program  delays  from  additional  development,  testing,  or 
other  prerequisite  events 

44% 

4.09 

2.35 

p22 

Defense  Contract  Audit  Agency  (DCAA)  administrative  actions 

44% 

3.92 

2.61 

F23 

Unplanned  Congressional  adds  to  Program  Baseline  (PB)  request 

43% 

3.90 

2.41 

F24 

Use  of  undefinitized  contract  action  delays 

42% 

3.73 

2.56 

F25 

Expenditure  contingent  on  hardware  delivery 

41% 

3.92 

2.41 

F26 

Loss  of  funding  through  reprogramming  action  to  higher  priority 
requirements  to  program  executive  officer  (PEO)  portfolio 

41% 

3.89 

2.46 

F27 

Lack  of  experience  levels  in  key  acquisition  functional  areas 

40% 

3.90 

2.30 

p28 

Awaiting  DAE-level  review  and  decision 

40% 

3.50 

2.42 

p29 

Shortage  of  cost  estimators 

40% 

3.67 

2.37 

p30 

Shortage  of  business/finance  personnel 

39% 

3.66 

2.32 

F31 

Programmatic  conflicts  between  government  and 
prime  contractor 

39% 

3.66 

2.32 

379 


Defense  ARJ,  October  2013,  Vol.  20 No.  3:373-400 


OSD’s  Obligation  and  Expenditure  Rate  Goals 


Factors  Rated  by  Adverse  Impact  Continued 

TB 

X 

a 

R32 

Preparing  Service  Acquisition  Executive/Component 

Acquisition  Executive  (SAE/CAE)-level  review  and  decision 

38% 

3.74 

2.02 

p33 

Delays  in  contractor  payment  due  to  late  invoices 

37% 

3.67 

2.35 

F34 

Unobligated  prior  year  funding  not  adequately  factored 

36% 

3.57 

2.23 

p35 

Component  Comptroller  Withhold 

35% 

3.58 

2.34 

p36 

Defense  Contract  Management  Agency  (DCMA) 
administrative  actions 

35% 

3.42 

2.36 

F37 

Redirection  of  contractor  efforts 

35% 

3.47 

2.23 

p38 

OSD  Comptroller  Withhold 

34% 

3.43 

2.37 

R39 

Shortage  of  technical/engineering/test  personnel 

34% 

3.51 

2.17 

p40 

Shortage  of  auditors 

33% 

3.17 

2.43 

F4' 

Slower  burn  rate  than  expected  due  to  unfavorable  Schedule 
Performance  Index 

33% 

3.25 

2.14 

F42 

Awaiting  SAE/CAE-level  review  and  decision 

32% 

3.33 

2.30 

[=43 

SAE/CAE/Component-directed  reprogramming 

32% 

3.27 

2.30 

p44 

Rescission 

32% 

3.16 

2.46 

F45 

Changes  in  systems  specs 

31% 

3.30 

2.03 

p46 

Tenure  of  program  manager  (PM)  and  others  in  key  positions 

31% 

3.11 

2.18 

F47 

Holding  award/incentive  fees  in  commitment  for 
future  obligation 

29% 

3.23 

2.35 

F48 

Inadequate  training 

29% 

3.29 

2.13 

p49 

Shortage  of  managers 

28% 

3.10 

2.17 

F50 

Insufficiently  planned  Overseas  Contingency  Operations 
(OCO)  funding 

27% 

3.07 

2.27 

F5' 

Shortage  of  staff 

26% 

2.99 

2.12 

F52 

Contractor  rework 

26% 

3.00 

2.14 

p53 

Deferred  payments  for  scheduling  earning  fees,  progress 
payments/performance-based  payments 

25% 

3.08 

2.20 

F54 

Effect  of  contract  type  on  outlay  rates 

24% 

2.99 

2.17 

p55 

Materiel/Systems  Command  Comptroller  Withhold 

24% 

2.71 

2.17 

p56 

Awaiting  PEO-level  review  and  decision 

24% 

2.80 

2.01 

F57 

Termination  liability 

22% 

2.72 

2.17 

p58 

Insufficient  workplace  tools/apps 

22% 

2.82 

2.01 

F59 

PEO-directed  programming 

21% 

2.83 

2.10 

p60 

Slower  burn  rate  than  expected  due  to  favorable 

Cost  Performance  Index 

21% 

2.77 

1.95 

F61 

PEO  Withhold 

20% 

2.39 

1.99 

F62 

Preparing  PEO-level  review  and  decision 

20% 

2.66 

1.53 

p63 

Production  line  issues 

19% 

2.82 

2.08 

p64 

Labor  disputes 

10% 

1.89 

1.64 

-la  =  25% 
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FIGURE  3.  IMPACT  RATINGS  ABOVE  X  IN  AGGREGATE 
DESCENDING  ORDER  WITH  RESPONDENT  GROUP  LOW 
AND  HIGH  RATINGS 
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to  reduce  the  adverse  effects  of  all  three  factors,  the  respondents  rec¬ 
ommended  the  “inclusion  of  more  risk  mitigation  into  contract  award 
planning,  more  realistic  timelines,  more  realistic  plans,  greater  funding 
stability,  reduction  in  bureaucratic  obstacles,  more  synchronized  inter¬ 
nal  processes,  and  better  aligned  accounting  systems.” 

Factors  Ranked  One  Standard  Deviation  Above 
the  Mean  (+1?) 

This  next  line  of  demarcation  (Figure  3,  factors  F'’-F9)  included  many 
contracting- related  factors  (i.e..  Shortage  of  contracting  officers  (F4), 
Contractor  proposal  prep  delays  (F6),  Request  for  Proposal  (RFP)  prep 
delays  (F8)  and  Source  selection  delays  (F9).  Nearly  all  the  factors  showed 
the  emergence  of  a  more  alarming  a  between  the  individual  respondent 
groups— as  high  as  18  percent  in  one  case  (i.e.,  Contractor  proposal  prep 
delays  [F6]).  For  this  particular  factor,  procurement  contracting  officers 
(PCO)  reported  the  highest  impact,  while  PMs  ranked  it  as  the  lowest. 
Senior  staff  cited  that  Shortage  of  contracting  officers  (F4)  created  the 
highest  impact,  while  PCOs  reported  it  had  the  lowest  impact.  With  a 
7  percent  a,  it  was  the  lowest  among  all  six  factors  in  this  grouping. 
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Given  that  six  of  the  top  nine  factors  were  contract-specific  factors 
that  ranked  above  +1  a  (Figure  3),  it  came  as  little  surprise  to  see  so  many 
reinforcing  comments  surface. 

•  “Lack  of  experienced  and  qualified  contract  specialists . . 

•  “Alarmingly  low  personnel  qualified . . .  many  unsure/lack 
guidance  and  experience ...” 

•  “Significantly  stressed  with  overtime  to  complete  all  con¬ 
tracting  actions  prior  to  close  of  fiscal  year.” 

•  “Inadequate  training . . .  inordinate  number  of  interns  with 
very  low  experience  in  all  career  fields.” 

•  “Lack  of  sufficient  legal  personnel  trained  in  Acquisition.” 

•  “Loss  in  brain  trust  and  skill  to  develop  complete,  clear 
SOWs  [Statements  of  Work]  using  proactive  contract 
language.” 

•  “SOW  writing  and  the  teaching  of  SOW  writing  classes  is 
greatly  left  to  contractors  or  support  contractors,  resulting 
in  unclear  language.” 

The  highest  frequency  of  occurrence  was  also  associated  with  con¬ 
tracting-related  factors  (Figure  3).  By  far,  the  aggregate  respondents 
rated  Shortage  of  contracting  officers  (F4)  as  the  single  highest  factor 
among  all  22  factors  measured  for  frequency.  Because  the  contracting 
activity  timeline  generally  has  lengthy  durations,  any  disruption  appears 
to  have  an  unmistakable  impact  on  contract  award.  Shortage  of  contract¬ 
ing  officers  (F4)  was  seen  as  having  the  most  significant  impact.  Several 
respondents  said  “multiple  contracting  actions  were  having  compound¬ 
ing  consequences.” 

The  two  remaining  factors  above  +1  a  Congressional  mark  (F5)  and 
OSD -directed  RMD  adjustment  (F7),  had  very  low  frequency  of  occur¬ 
rences,  but  still  reported  a  very  high  impact  similar  to  CRA.  When 
combining  these  with  F4,  all  three  appear  to  be  a  strong  antecedent  force 
(or  moderating  factor)  to  the  already  time-consuming  chain  of  contract¬ 
ing  actions. 
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Factors  Ranked  Above  x 

This  final  grouping  (Figure  3,  factors  F10-F31)  accounted  for  the 
remaining  22  impact  factors.  Perception  polarities  persisted,  especially 
between  two  respondent  groups— senior  staff  outside  the  program  office 
and  PMs  inside  program  offices.  For  PMs  in  every  case  except  one  (i.e.. 
Component-directed  POM  adjustment  [F17]),  the  impact  factors  ranked 
well  below  x.  In  sharp  contrast,  senior  staff,  in  every  case  except  one 
(i.e..  Component-directed  POM  adjustment  [F17]),  stated  the  majority 
of  the  top  31  factors  had  the  largest  impact— or  close  to  it— among  all 
respondent  groups. 

Even  though  the  remaining  impact  factors  above  x  are  still  sig¬ 
nificant,  the  researchers  shifted  the  focus  to  the  presence  of  any  strong 
correlations  since  factor  couplings  could  be  having  a  moderating  effect 
and  require  a  closer  look. 

Factors  That  Correlate 

Table  3  summarizes  the  strongest  and  weakest  factor  correlations 
for  all  respondents  queried.  Several  strong  correlations  surfaced  for 
factors  above  3c.  Changes  in  user  requirements  (F11)  and  Changes  in  user 
priorities  (F19)  were  very  strongly  correlated.  In  three  specific  instances, 
two  factors  above  3c  were  very  strongly  correlated  with  three  factors 
that  fell  below  3c:  Lack  of  experience  levels  in  key  acquisition  functional 
areas  (F27)  and  Inadequate  training  (F48);  Lack  of  experience  levels  in 
key  acquisition  functional  areas  (F27)  and  Tenure  of  PM  and  others  in 
key  positions  (F46);  and  DCMA  administrative  actions  (F36)  and  DCAA 
administrative  actions  (F22).  Three  contract-related  factors  (F4,  F8,  and 
F9)  showed  weaker  correlations  than  expected.  Whether  a  factor  had 
a  weak  correlation  doesn’t  mean  it  had  any  less  importance,  but  any 
course  of  action  intended  to  mitigate  the  presence  of  any  impact  factor 
strongly  correlated  with  another  should  be  weighed  more  heavily  in  any 
recommended  action.  For  example,  the  turnover  of  PMs  could  be  part  of 
the  experience  quotient. 

Factor  Plotting 

The  researchers  generated  a  scatter  plot  diagram  (Figure  4)  that 
punctuated  how  the  31  factors  fluctuated  between  impact  and  frequency 
of  occurrence.  In  some  cases,  the  impact  of  certain  factors  occurred  with 
low  frequencies  of  occurrence.  In  other  cases,  the  frequency  of  occur¬ 
rence  compounded  the  impacts. 
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TABLE  3.  FACTOR  CORRELATION  COUPLING 

Strongest  Correlation  Weakest 

r  r2  Coefficients  Correlation 


Experience,  Training,  and  Tenure: 


.84 

71% 

F27  Key  acquisition  experience 
levels  and  F48  Inadequate  training 

.78 

61% 

F27  Key  acquisition  experience 
levels  and  F46  Tenure  of  PM  and 
other  key  positions 

Administrative  Actions: 

.81 

76% 

F36  DCMA  and  F22  DCAA 

.82 

67% 

F"  User  requirements  and 

F19  User  priorities 

.70 

49% 

F19  User  priorities  and 

F13  Stakeholder  requirements 

Contract-related  Activities: 

.71 

50% 

F6  Contractor  proposal  delay  and 

F2  Contract  negotiations  delays 

.70 

49% 

F3  Contract  award  delays  and 

F2  Contract  negotiations  delays 

The  higher  the  %,  the  stronger  the  direction  and 
strength  of  the  linear  relationship  between  the 
variables 


Factors  #  1  -  3  >  +2o; 
Factors  #  4  -  9  >  +1o 
Factors  #  10  -  31  >  x 


F1  Late  release  of  full  obligation/budget 
authority  due  to  CRA 

F4  Shortage  of  contracting  officers 

F5  Congressional  mark/rescission 

F7  OSD-directed  RMD  adjustment 

F8  RFP  prep  delays 

F'°  Unrealistic/overly  optimistic  spend 
plans 

F12  Changes  to  program  acquisition 
strategy 

F15  Lack  of  decision  authority 

F16  Implementation  of  new  OSD/Service 
policy 

F17  Component-directed  POM  adjustment 

F18  Awaiting  reprogramming  action 

F20Realistic  spend  plans,  but  risks 
materialized 

F21  Program  delays  from  prerequisite 
events 

F23  Unplanned  Congressional  adds  to  PB 
request 

F25 Expenditure  contingent  on  hardware 
delivery 

F26Loss  of  funding  through  reprogramming 

F29Shortage  of  cost  estimators 

F30Shortage  of  business/finance  personnel 

F31  Programmatic  conflicts  between 
government  and  prime  contractor 


The  research  data  results  were  rebased  to  a  Likert-like  scale  for 
plotting  the  frequency  and  adverse  impact  response  averages.  The 
researchers  included  Factors  F29-F31  in  Figure  4  because  they  only  fall 
slightly  below  x. 

For  the  relationships  that  were  co-linear  (e.g.,  the  most  strongly  cor¬ 
related  depicted  in  Table  3),  the  researchers  explored  whether  they  also 
behaved  as  strong  predictors  across  the  sample  population.  After  inves¬ 
tigating  t-ratios  (used  with  ACAT  Level  factors)  and  beta-weights  (used 
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for  the  sample  population),  the  researchers  determined  the  relation¬ 
ships  were  not  significantly  co-linear  enough  to  substantiate  causation. 
Consequently,  there  was  no  merit  in  running  any  further  regression  that 
analyzed  the  factors  as  predictors.  However,  the  researchers  conducted 
another  set  of  tests  by  modulating  certain  respondent  demographics  and 
holding  constant. 


FIGURE  4.  SCATTER  PLOT  OF  IMPACT  FACTORS  WITH  FREQUENCY 


Scatter  Plot  Results  >  x  (for  Impact) 


30 

4 

29 

31 

16  11 
19  13 

8  6 

““ 2 

3 

24 

2 

21 

1  18 

1 

17 

7 

5 

3.5  4.0  4.5  5.0  5.5 

IMPACT  T.j| 


>  *la 

>  +1ct 
>+X 

1  Factor  listed  by  rating  of 
adverse  Impact,  but  not 
plotted  by  frequency. 
Frequency  data  not 
collected. 


1  Late  release  of  full  obligation/budget  authority  due  to 
CRA 

2  Contract  negotiations  delays 

3  Contract  award  delays 
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5  Congressional  mark 

6  Contractor  proposal  prep  delays 

7  OSD-directed  RMD  adjustment 

8  RFP  prep  delays 

9  Source  selection  delays 

70  Unrealistic/overly  optimistic  spend 
plans’ 

11  Changes  in  user  requirements 

12  Changes  to  program  acquisition 
strategy' 

13  Changes  in  other  stakeholder  requirements 

14  Preparing  DAE-ievei  review  and 
decision’ 

15  Lack  of  decision  authority  at 
expected  levels’ 

16  Implementation  of  new  OSD/Service  policy 

17  Component-directed  POM  adjustment 

18  Awaiting  reprogramming  action 

19  Changes  in  user  priorities 

20  Realistic  spend  plans,  but  risks 
materialized’ 

21  Program  delays  from  additional  development, 
testing,  or  other  prerequisite  events 

22  DCAA  administrative  actions’ 

23  Unplanned  Congressional  adds  to  PB  request 

24  Use  of  undefinitized  contract  action  delays 

25  Expenditure  contingent  on 
hardware  delivery’ 

26  Loss  of  funding  through  reprogramming  action  to 
higher  priority  requirements  to  PEO  portfolio 

27  Lack  of  experience  levels  in  key 
acquisition  functional  areas’ 

28  Awaiting  DAE-ievei  review  and 
decision’ 

29  Shortage  of  cost  estimators 

30  Shortage  of  business/finance  personnel 

31  Programmatic  conflicts  between  government  and 
prime  contractor 
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Factor  Plotting— Modulating  ACAT  Levels 

Figure  5  shows  how  the  factor  rankings  changed  after  isolating 
ACAT  levels. 


FIGURE  5.  FACTOR  RATINGS  >  x  ACAT  LEVEL 


31 

11  ql3 

24  21 


16 


1  ,75 


2.0-1 - 

3.8  4.3  4.8  5.3  5.8 

Impact 


ACAT  II 

n  = 

28 

ACAT  III 

n 

=  23 

5.0- 

51  2? 

50 

39 

4 

7  . 

4.0“ 

7  Cl— 

37 

6  8 

w : 

13 

9 

26 

3 

19 

1 

17 

1 

3.U 

2  0— 

7 

17 

5 

8  4 .1 

!  4.1 

Imp 

5  5.: 

act 

3  5. 

8 

■3.8 

■/ 

!  4.i 

:  4.i 

Imp 

3  5.: 

act 

3  5j 

8 

MODULATION  OF  FACTORS 

Factor  Decrease: 

F!  Awaiting  reprogramming  action 

F9  Changes  in  user  priorities 

FB  Unplanned  Congressional  adds  to  PB 
request 

F*  Loss  of  funding  through 
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priority  requirements  to  PEO  portfolio 

Factor  Increase: 

P  Programmatic  conflicts  between 
government  and  prime  contractor 


MODULATION  OF  FACTORS 

Factor  Decrease: 

F2  Contract  negotiation  delays 
F!  Contract  award  delays 
F*  Shortage  of  contracting  officers 
Ff  Contractor  proposal  prep  delays 
P  OSD-directed  RMD  adjustment 
F*  RFP  prep  delays 
F9  Source  selection  delays 
F"  Changes  in  user  requirements 
FB  Changes  in  other  stakeholder 
requirements 

F16  Implementation  of  new  OSD/Service 
policy 

F"  Awaiting  reprogramming  action 
F19  Changes  in  user  priorities 
F21  Program  delays  from  prerequisite 
events 

F23  Unplanned  Congressional  adds  to  PB 
request 

F24  Use  of  undefinitized  contract  action 
delays 


MODULATION  OF  FACTORS 

Factor  Decrease: 

Fl  Implementation  of  new  OSD/Service 
policy 

F‘  Awaiting  reprogramming  action 

F9  Changes  in  user  priorities 

F21  Program  delays  from  prerequisite 
events 

F2i  Unplanned  Congressional  adds  to  PB 
request 

F2<  Use  of  undefinitized  contract  action 
delays 

Factor  Increase: 

F29  Shortage  of  Cost  Estimators 

F!"  Shortage  of  business/finance 
personnel 

F  Redirection  of  contractor  efforts 

F!S  Shortage  of  technical/engineering/ 
testing  personnel 

P  Shortage  of  staff 


ACAT  I.  Funding  and  requirements  factors  (F18,  F19,  F23,  and  F28) 
previously  ranked  above  x  dropped  below  x  while  Contractor  proposal 
prep  delays  (F6)  rose  markedly  to  become  the  highest  impact  factor. 
Component-directed  POM  adjustment  (F17)  made  a  noticeable  shift  to 
the  top  nine  factors  (or  one  standard  deviation  above  the  mean). 
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ACAT  II.  Fifteen  of  the  factors  previously  ranked  above  x  dropped 
below  x  (leaving  only  F1,  F2,  F3,  and  F17).  Four  of  the  factors  that  fell  below 
x  included  contracting-related  factors  (F4,  F6,  F8,  and  F9). 

ACAT  III.  Six  of  the  factors  (F16,  F18,  F19,  F21,  F23,  and  F24)  previously 
ranked  above  x  dropped  below  x.  Shortages  of  personnel  (F29,  F30,  F39,  and 
F51)  and  Redirection  of  contractor  efforts  (F37)  became  more  dominat¬ 
ing  issues  for  the  respondents.  Changes  in  user  priorities  (F19),  Changes 
in  other  stakeholder  requirements  (F13),  and  Loss  of  funding  through 
reprogramming  action  to  higher  priority  requirements  to  PEO  portfolio 
(F26)  all  moved  significantly  above  x. 

What  does  this  mean?  The  more  detailed  differentiation  seen  in  the 
scatter  plots  gives  additional  insight  into  the  factors  that  would  benefit 
from  a  more  focused  investigation  of  each  ACAT.  In  some  cases,  reduc¬ 
ing  frequency  of  occurrence  or  perhaps  instituting  more  early  warning 
metrics  could  have  a  marked  effect  in  reducing  any  adverse  impacts. 

Factor  Plotting— Modulating  Service  Components  and  DoD 

Figure  6  shows  how  the  factor  rankings  changed  after  isolating 
Service  Components. 

U.S.  Army.  No  factors  fell  below x.  The  only  component  where 
factors  moved  above  x  was  Shortage  of  auditors  (F40)  and  Insufficiently 
planned  Overseas  Contingency  Operations  (OCO)  funding  (F50).  Based  on 
historical  information,  OCO  funding  will  most  likely  continue  to  present 
challenges  since  contingency  funding  needs  are  less  predictable  during 
a  wartime  footing. 

U.S.  Air  Force.  Shortage  of  contracting  officers  (F4)  and  Use  of 
undefinitized  contract  action  delays  (F24)  both  dropped  below  x.  Even 
though  Shortage  of  contracting  officers  moved,  there  were  no  companion 
drops  in  contracting-related  factors. 

U.S.  Navy.  Six  factors  dropped  below  x.  Implementation  of  new 
OSD/Service  Policy  (F16),  Awaiting  reprogramming  action  (F18),  Changes 
in  user  priorities  (F19),  Unplanned  Congressional  adds  to  PB  request  (F23), 
Use  of  undefinitized  contract  action  delays  (F24),  and  Loss  of  funding 
through  reprogramming  action  to  higher  priority  requirements  to  PEO 
portfolio  (F26)  became  less  of  an  impact.  For  Navy  respondents,  there  was 
no  notable  movement  in  the  top  six  contracting-related  factor  collective. 
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FIGURE  6.  FACTOR  RATINGS  >x  BY  COMPONENT 
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DoD.  Three  factors  fell  below  x  (i.e.,  Implementation  of  new  OSD/ 
Service  policy  [F16],  Component-directed  POM  adjustment  [F17],  and 
Use  of  undefinitized  contract  action  delays  [F24]),  while  three  factors 
rose  above  x:  OSD  Comptroller  Withhold  (F38),  Shortage  of  business/ 
finance  personnel  (F30),  and  Shortage  of  technical/engineering/test 
personnel  (F39). 

What  does  this  mean?  The  Army  was  the  only  one  of  the  four  group¬ 
ings  that  was  significantly  affected  by  Use  of  undefinitized  contract 
action  delays  (F24);  and  DoD  was  the  only  one  of  the  four  groupings  that 
was  significantly  affected  by  OSD  Comptroller  Withhold  (F38),  Shortage 
of  business/ finance  personnel  (F30),  and  Shortage  of  technical/engineer¬ 
ing/test  personnel  (F39). 

Factor  Plotting— Modulating  Respondent  Groups 

Figure  7  shows  how  the  factor  rankings  changed  after  isolating  the 
respondent  groups. 

Program  Office.  Six  factors  dropped  below x:  Awaiting  repro¬ 
gramming  action  (F18),  Changes  in  user  priorities  (F19),  Program  delays 
from  additional  development,  testing,  or  other  prerequisite  events  (F21), 
Unplanned  Congressional  adds  to  PB  request  (F23),  Use  of  undefinitized 
contract  action  delays  (F24),  and  Loss  of  funding  through  reprogramming 
action  to  higher  priority  requirements  to  PEO  portfolio  (F26).  No  factors 
fell  below  x. 

PEO.  Use  of  undefinitized  contract  action  delays  (F24)  fell  below 
x,  while  four  factors  rose  above  3c:  Shortage  of  cost  estimators  (F29), 
Shortage  of  business/finance  personnel  (F30),  Component  Comptroller 
Withhold  (F35),  and  Insufficiently  planned  OCO  funding  (F50). 

Senior  OSD  Staff.  Awaiting  reprogramming  action  (F18)  fell  below 
x  while  13  factors  rose  above  3c. 

For  PEO  and  senior  OSD  staff,  personnel  shortages  (F29,  F30,  F20,  and 
F40)  became  more  dominant  while  Awaiting  reprogramming  action  (F18) 
became  less  dominant  for  program  office  and  senior  OSD  staff  person¬ 
nel.  Of  the  three  groupings  in  this  particular  case,  nowhere  were  there 
more  factor  increases  than  for  senior  OSD  staff  personnel.  The  rise  in 
Unobligated  prior  year  funding  not  adequately  factored  (F34),  SAE/CAE/ 
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FIGURE  7.  FACTOR  RATINGS  >  JC  WHEN  NEGOTIATED 
CONTRACT  COSTS  WERE  LOWER  THAN  COSTS  PROJECTED 
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Component-directed  reprogramming  (F43),  and  PEO-directed  program¬ 
ming  (F59)  seemed  intuitive  since  senior  staff  may  see  first-hand  the 
longer  time  it  takes  for  program  managers  to  react  to  changes  in  their 
plans.  However,  it  was  very  interesting  to  note  the  disparities  between 
how  senior  OSD  staff  personnel  responded  to  survey  queries  regarding 
the  major  impediments  to  meeting  OSD’s  Obligation  and  Expenditure 
rate  goals  versus  the  responses  from  program  office  personnel,  especially 
shortage  of  personnel  and  contract-specific  factors  (i.e..  Changes  in  sys¬ 
tems  specs  (F45)  and  Redirection  of  contractor  efforts  (F37).  What  does 
this  mean?  This  wide  perception  disparity  deserves  a  more  intensive 
understanding  since  it  could  be  creating  false  perceptions  that  could 
lead  to  misrepresented  positions  and  even  unsubstantiated  decisions. 
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Factor  Plotting— Modulating  Program  Phase 
and  Cost  Projections 

Figure  8  shows  how  the  factor  rankings  changed  after  modulating  by 
program  phase  when  their  negotiated  contract  costs  were  significantly 
lower  than  projections. 


FIGURE  8.  FACTOR  RATINGS  >x  BY  POSITION 
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Factor  Decrease: 

F"  Awaiting  reprogramming  action 

F®  Changes  in  user  priorities 

Fa  Program  delays  from  prerequisite 
events 

FB  Unplanned  Congressional  adds  to  PB 
request 

F24  Use  of  undefinitized  contract  action 
delays 

F®  Loss  of  funding  through  reprogram¬ 
ming  action  to  higher  priority 
requirements  to  PEO  portfolio 


Factor  Decrease: 

F24  Use  of  undefinitized  contract  action 
delays 

Factor  Increase: 

F29  Shortage  of  cost  estimators 
F50  Shortage  of  business/finance  personnel 
F®  Component  Comptroller  Withhold 
F50  Insufficiently  planned  0C0  funding 


Factor  Decrease: 

Fl!  Awaiting  reprogramming  action 
Factor  Increase: 

F29  Shortage  of  cost  estimators 
F51  Shortage  of  business/finance  personnel 
FH  Unobligated  prior  year  funding  not 
adequately  factored 
F®  Redirection  of  contractor  efforts 
F®  Shortage  of  technical/engineering/ 
testing  personnel 
F40  Shortage  of  auditors 
F4!  SAE/CAE/Component-directed 
reprogramming 
F44  Rescission 

F®  Changes  in  systems  specs 
F42  Holding  award/incentive  fees  in 
commitment  for  future  obligation 
F5"  Insufficiently  planned  0C0  funding 
F57  Termination  liability 
F59  PEO-directed  programming 


Development  Phase  (Technology  Development  [TD]  and 
Engineering,  Manufacturing,  and  Development  [EMD]).  Four 
factors  dropped  below  x,  including  Changes  in  other  stakeholder 
requirements  (F13),  Awaiting  reprogramming  action  (F18),  Unplanned 
Congressional  adds  to  PB  request  (F23),  and  Loss  of  funding  through 
reprogramming  action  to  higher  priority  requirements  to  PEO  portfolio 
(F26).  Four  factors  rose  above  x,  including  Shortage  of  business/finance 
personnel  (F30),  Programmatic  conflicts  between  government  and  prime 
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contractor  (F31),  Shortage  of  technical/engineering/test  personnel  (F39), 
and  Holding  award/incentive  fees  in  commitment  for  future  obligation 
(F47).  In  two  cases.  Programmatic  conflicts  between  government  and 
prime  contractor  (F31)  and  Implementation  of  new  OSD  policy  (F16)  made 
a  noticeable  shift  to  the  top  nine  factors  (or  one  standard  deviation  above 
the  mean). 

Procurement  Phase  (Low  Rate  Initial  Production  [LRIP]  and 
Full  Rate  Production  [FRP]).  Eight  of  the  factors  that  previously 
ranked  above  3c  dropped  below  3c.  The  majority  of  the  movement  was 
seen  in  factors  involving  program  delays,  and  funding  and  require¬ 
ments  changes.  The  factors  involving  program  delays  included  Program 
delays  from  additional  development,  testing,  or  other  prerequisite  events 
(F21),  and  Use  of  undefinitized  contract  action  delays  (F24).  The  factors 
involving  funding  delays  included  Unplanned  Congressional  adds  to  PB 
requests  (F23),  and  Awaiting  reprogramming  action  (F18).  The  factors 
involving  requirements  changes  included  Changes  in  user  requirements 
(F11),  Changes  in  other  stakeholder  requirements  (F13),  Changes  in  user 
priorities  (F19),  and  Loss  of  funding  through  reprogramming  action  to 
higher  priority  requirements  to  PEO  portfolio  (F26).  Both  Unobligated 
prior  year  funding  not  adequately  factored  (F34)  and  Shortage  of  techni¬ 
cal,  engineering,  and  test  personnel  (F39)  rose  above  3c. 

In  both  phases,  Changes  in  other  stakeholder  requirements  (F13), 
Awaiting  reprogramming  action  (F18),  Unplanned  Congressional  adds 
to  PB  requests  (F23),  and  Loss  of  funding  through  reprogramming  action 
to  higher  priority  requirements  to  PEO  portfolio  (F28)  fell  below 3c.  In  the 
context  of  modulating  by  program  phase,  the  researchers  found  that  any 
factor  movement  was  negligible  when  costs  met  or  exceeded  projections. 

What  does  this  mean?  Changes  in  user  requirements  (F11)  could 
potentially  be  more  stable  during  the  production  phase  and  no  longer 
become  a  factor.  However,  the  emergence  of  Programmatic  conflicts 
between  government  and  prime  contractor  (F31)  during  the  development 
phase  could  perhaps  be  the  sign  of  competing  motivations  between  DoD 
and  industry  as  well  as  more  prominent  technical  and  schedule  risks. 
All  three  could  result  in  programmatic  delays. 
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Respondent  Comments  Regarding  the  Factors 

The  respondents  were  also  asked  several  open-ended  questions  about 
whether  they  found  the  use  of  metrics  helpful  in  better  meeting  OSD 
goals  as  well  as  any  process  improvements  they  would  recommend.  They 
stated  the  metrics  making  a  difference  for  them  included  “real-time 
monitoring,  frequent  reviews,  tight  coupling  to  contractor  actions  and 
milestones,  and  realistic  spend  plans.”  When  asked  about  any  necessary 
improvements  to  current  processes,  the  respondents  recommended 
including  a  CRA  duration  variable  that  readjusted  expectations, 
establishing  more  realistic  program  goals,  ensuring  more  funding 
stability,  reducing  bureaucratic  obstacles  and  streamlining  more 


FIGURE  9.  SAMPLING  OF  RESPONDENT  COMMENTS 


“Takes  too  long  to  get  Acquisition  Strategies  and  Acquisition 
Plans  written  and  approved." 

“Personnel  do  not  have  experience  with  the  subject  matter." 


“Inadequate  proposals,  protracted  negotiations,  lengthy 
audits,  and  lengthy  pre-award  processes.” 


“ Had  to  defer/re-prioritize  requirements  execution  and  carry 
forward  funding  to  cover  cutbacks/shortfall." 

“Changes  in  requirements  precipitated  by  other 
stakeholders'  actions  and  ill-defined  requirements.  ” 

“User  leadership  routinely  changes  requirement  &  priorities." 

“MiPR  [Military  Interdepartmental  Purchase  Request]  billing 
process  can  delay  expenditures  from  90  to  120  days." 

“Delays  in  negotiating  best  deal  for  gov't  and  sometimes 
delays  in  getting  acceptable  proposals.” 


“Extensive  reviews,  too  long  to  get  decision  briefs  through 
oversight  layers— not  always  value  added.  ” 

“Multiple  instances  where  milestone  documentation  took 
upwards  of  9  months  to  a  year  to  get  approved." 


"The  problem  isn't  unrealistic  or  overly  optimistic  spend 
plans  as  much  as  it's  not  knowing  when  funds  will  be 
appropriated  and  how  much  will  be  apportioned  by  the 
executing  organization." 
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outdated  processes,  forging  greater  cooperation  between  government 
and  industry,  and  synchronizing  disparate  accounting  systems  used  in 
Obligation  and  Expenditure  reporting. 

The  respondents  provided  a  number  of  additional  qualitative  com¬ 
ments  that  reinforced  the  quantitative  data,  especially  for  the  factors 
above  >  x  that  were  causing  obligation  rate  interference. 

Recommendations 

What  next?  Based  on  the  research  findings  presented  in  this  article, 
a  number  of  impact  factors  above  x,  if  sufficiently  addressed,  could  help 
lower  the  barriers  to  the  attainment  of  OSD’s  Obligation  and  Expenditure 
rate  goals.  Hence,  the  researchers  offer  the  following  recommendations: 

•  Institute  an  Obligation  and  Expenditure  baseline  adjust¬ 
ment  for  programs  affected  by  any  funding  delay  or 
limitation  (especially  CRA),  then  measure  a  program’s  prog¬ 
ress  to  that  revised  adjustment. 

•  More  thoroughly  review  the  entire  contracting  action  value 
chain.  Look  closely  at  efficiency  opportunities  along  the 
review  and  decision  cycle  continuum,  especially  from  the 
time  an  RFP  is  developed  to  the  time  a  contract  is  let.  Set 
reasonable  time  thresholds  with  triggers  that  afford  more 
proactive  measures  by  PMs— and  confirm  productivity. 

•  Establish  a  recurring  communication  forum  among  key 
stakeholders,  especially  PMs  and  OSD,  to  dialogue  more 
frequently  and  eliminate  perception  gaps  that  could  be 
creating  counterproductive  actions  and  misconceptions. 

•  Track  requirement  changes  throughout  a  program’s  life  and 
look  more  strategically  at  the  effects  on  program  execution 
and  accompanying  Acquisition  Program  Baselines.  Despite 
ACAT  levels,  an  obvious  ripple  effect  is  associated  with  any 
substantive  change  in  program  content  across  a  program’s 
life  that  should  be  codified  more  comprehensively.  However, 
there  are  also  issues  associated  with  different  ACAT  levels, 
which  must  be  noted. 
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Review  the  program  review  cycle  and  streamline  wherever 
possible.  Checks  and  balances  within  the  DoD’s  acquisition 
community  have  always  been  a  vital  constituent  component 
of  program  execution,  but  every  review  should  have  a  dis¬ 
tinctive  purpose,  exit  criteria,  and  associated  suspense  date 
that  is  just  as  material  and  credible. 

Build  and  maintain  realistic  spend  plans,  measure  against 
them,  account  for  contingencies,  and  make  adjustments 
with  required  frequency  due  to  real  world  realities.  Since 
spend  plans  are  subjected  to  so  many  real  world  program¬ 
matic  eventualities,  updating  them  is  vital.  Collaborate  with 
senior  leadership  early  enough  about  required  adjustments 
to  avoid  more  draconian  measures  later. 

Validate  the  key  personnel  shortage  areas  and  recognize  the 
time  it  takes  to  rebuild  those  experience  levels. 

Nurture  experience  in  key  functional  areas  with  strong 
catalysts  such  as  disciplined  on-the-job  training,  programs, 
mentoring,  and  guidance.  With  the  recent  surge  of  contract¬ 
ing  specialist  interns,  their  progress  as  a  group  should  be 
measured  more  carefully. 

Evaluate  the  real  effects  of  reprogramming  action  or 
realignment  of  future  budget  decisions  before  any  correc¬ 
tive  action  is  taken. 

Conduct  a  wholesale  review  of  the  program  execution  met¬ 
rics  currently  in  place  and  determine  their  usefulness  and 
effectiveness.  What  are  they  actually  measuring?  How  are 
these  data  (metrics)  used  and  are  they  worth  collecting? 
Consolidate  whenever  practical  and  eliminate  the  data 
(metrics)  that  have  outlived  their  usefulness. 

Encourage  innovation  and  avoid  the  “bookkeeping  process” 
as  RAND  Corporation  found  in  a  recent  study  that  could  be 
limiting  improvements  championed  by  PMs  (Blickstein  & 
Nemfakos,  2009). 


Defense ARJ,  October 3013,  Vol.  20 No.  3: 373-400 


396 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


Summary 

On  Feb.  5,  2013,  the  authors  presented  the  study  results  discussed 
in  this  article  to  Assistant  Secretary  of  Defense  for  Acquisition 
Katrina  McFarland  and  other  key  OSD  senior  staff.  With  the  metrics 
she  plans  to  institute  with  Better  Buying  Power  (BBP)  2.0,  DoD  will 
have  another  means  to  address  many  of  the  impact  factors  discussed 
herein  and  a  host  of  other  variables  that  could  be  encumbering  pro¬ 
gram  execution  expectations. 

On  Sept.  10,  2012,  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics  Frank  Kendall,  and  Under  Secretary  of 
Defense  (Comptroller)  Robert  F.  Hale,  jointly  signed  a  memorandum 
that  listed  six  tenets  that  could  help  combat  some  of  the  same  factors 
discussed  in  this  article  regarding  the  disposition  of  DoD’s  unobligated 
funds  (DoD,  2012).  Over  time,  realization  of  these  tenets  might  also 
reduce  perception  disparity  gaps  among  the  key  personnel  that  have  a 
hand  in  ensuring  our  warfighters  continue  to  get  the  weapon  systems 
they  need— and  on  time— to  best  support  our  national  military  strategy. 
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